Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[libc] newheadergen: script adjusted for cmake #98825

Merged
merged 8 commits into from
Jul 16, 2024

Conversation

aaryanshukla
Copy link
Contributor

  • added entrypoints and headerfile parameters depending on target
  • fixed nits in yaml files causing errors
  • tested with new cmake config
  • cmake patch will be seperate

@llvmbot llvmbot added the libc label Jul 14, 2024
@llvmbot
Copy link
Collaborator

llvmbot commented Jul 14, 2024

@llvm/pr-subscribers-libc

Author: None (aaryanshukla)

Changes
  • added entrypoints and headerfile parameters depending on target
  • fixed nits in yaml files causing errors
  • tested with new cmake config
  • cmake patch will be seperate

Full diff: https://github.com/llvm/llvm-project/pull/98825.diff

6 Files Affected:

  • (added) libc/newhdrgen/header2.py (+96)
  • (added) libc/newhdrgen/yaml/features.yaml (+8)
  • (modified) libc/newhdrgen/yaml/pthread.yaml (+3-28)
  • (modified) libc/newhdrgen/yaml/sys_random.yaml (+1-1)
  • (modified) libc/newhdrgen/yaml/time.yaml (+1-1)
  • (modified) libc/newhdrgen/yaml_to_classes.py (+64-27)
diff --git a/libc/newhdrgen/header2.py b/libc/newhdrgen/header2.py
new file mode 100644
index 0000000000000..e115a7e8ef358
--- /dev/null
+++ b/libc/newhdrgen/header2.py
@@ -0,0 +1,96 @@
+#!/usr/bin/env python
+#
+# ===- HeaderFile Class for libc function headers  -----------*- python -*--==#
+#
+# Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+# See https://llvm.org/LICENSE.txt for license information.
+# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+#
+# ==-------------------------------------------------------------------------==#
+
+
+class HeaderFile:
+    def __init__(self, name):
+        self.name = name
+        self.macros = []
+        self.types = []
+        self.enumerations = []
+        self.objects = []
+        self.functions = []
+        self.includes = []
+
+    def add_macro(self, macro):
+        self.macros.append(macro)
+
+    def add_type(self, type_):
+        self.types.append(type_)
+
+    def add_enumeration(self, enumeration):
+        self.enumerations.append(enumeration)
+
+    def add_object(self, object):
+        self.objects.append(object)
+
+    def add_function(self, function):
+        self.functions.append(function)
+
+    def add_include(self, include):
+        self.includes.append(include)
+
+    def __str__(self):
+        content = []
+
+        content.append(
+            f"//===-- C standard declarations for {self.name} ------------------------------===//"
+        )
+        content.append("//")
+        content.append(
+            "// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions."
+        )
+        content.append("// See https://llvm.org/LICENSE.txt for license information.")
+        content.append("// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception")
+        content.append("//")
+        content.append(
+            "//===----------------------------------------------------------------------===//\n"
+        )
+
+        header_guard = f"__LLVM_LIBC_DECLARATIONS_{self.name.upper()[:-2]}_H"
+        content.append(f"#ifndef {header_guard}")
+        content.append(f"#define {header_guard}\n")
+
+        content.append("#ifndef __LIBC_ATTRS")
+        content.append("#define __LIBC_ATTRS")
+        content.append("#endif\n")
+
+        content.append("#ifdef __cplusplus")
+        content.append('extern "C" {')
+        content.append("#endif\n")
+
+        for include in self.includes:
+            content.append(str(include))
+
+        for macro in self.macros:
+            content.append(f"{macro}\n")
+
+        for type_ in self.types:
+            content.append(f"{type_}")
+
+        if self.enumerations:
+            combined_enum_content = ",\n  ".join(
+                str(enum) for enum in self.enumerations
+            )
+            content.append(f"\nenum {{\n  {combined_enum_content},\n}};")
+
+        for function in self.functions:
+            content.append(f"{function} __LIBC_ATTRS;\n")
+
+        for object in self.objects:
+            content.append(f"{object} __LIBC_ATTRS;\n")
+
+        content.append("#ifdef __cplusplus")
+        content.append("}")
+        content.append("#endif\n")
+
+        content.append(f"#endif")
+
+        return "\n".join(content)
diff --git a/libc/newhdrgen/yaml/features.yaml b/libc/newhdrgen/yaml/features.yaml
new file mode 100644
index 0000000000000..86bc0acfe89ed
--- /dev/null
+++ b/libc/newhdrgen/yaml/features.yaml
@@ -0,0 +1,8 @@
+header: features.h
+standards: 
+  - stdc
+macros: []
+types: []
+enums: []
+objects: []
+functions: []
diff --git a/libc/newhdrgen/yaml/pthread.yaml b/libc/newhdrgen/yaml/pthread.yaml
index 14a562082d5de..f22767eb1b752 100644
--- a/libc/newhdrgen/yaml/pthread.yaml
+++ b/libc/newhdrgen/yaml/pthread.yaml
@@ -13,33 +13,8 @@ types:
   - type_name: __pthread_start_t
   - type_name: __pthread_once_func_t
   - type_name: __atfork_callback_t
-enums:
-  - name: PTHREAD_CREATE_JOINABLE
-    value: 0x0
-  - name: PTHREAD_CREATE_DETACHED
-    value: 0x1
-  - name: PTHREAD_MUTEX_NORMAL
-    value: 0x0
-  - name: PTHREAD_MUTEX_ERRORCHECK
-    value: 0x1
-  - name: PTHREAD_MUTEX_RECURSIVE
-    value: 0x2
-  - name: PTHREAD_MUTEX_DEFAULT
-    value: 0x0
-  - name: PTHREAD_PROCESS_PRIVATE
-    value: 0x0
-  - name: PTHREAD_PROCESS_SHARED
-    value: 0x1
-  - name: PTHREAD_MUTEX_STALLED
-    value: 0x0
-  - name: PTHREAD_MUTEX_ROBUST
-    value: 0x1
-  - name: PTHREAD_RWLOCK_PREFER_READER_NP
-    value: 0
-  - name: PTHREAD_RWLOCK_PREFER_WRITER_NP
-    value: 1
-  - name: PTHREAD_RWLOCK_PREFER_WRITER_NONRECURSIVE_NP
-    value: 2
+  - type_name: pthread_rwlock_t
+enums: []
 functions:
   - name: pthread_atfork
     standards: 
@@ -184,7 +159,7 @@ functions:
   - name: pthread_exit
     standards: 
       - POSIX
-    return_type: __Noreturn void
+    return_type: _Noreturn void
     arguments:
       - type: void *
   - name: pthread_getname_np
diff --git a/libc/newhdrgen/yaml/sys_random.yaml b/libc/newhdrgen/yaml/sys_random.yaml
index 233fb2c7988cb..6d84056d7dd71 100644
--- a/libc/newhdrgen/yaml/sys_random.yaml
+++ b/libc/newhdrgen/yaml/sys_random.yaml
@@ -4,7 +4,7 @@ types:
   - type_name: ssize_t
   - type_name: size_t
 enums: []
-objects:
+objects: []
 functions:
   - name: getrandom
     standards: 
diff --git a/libc/newhdrgen/yaml/time.yaml b/libc/newhdrgen/yaml/time.yaml
index e7f8de65eeb75..220d4328dbbdb 100644
--- a/libc/newhdrgen/yaml/time.yaml
+++ b/libc/newhdrgen/yaml/time.yaml
@@ -15,7 +15,7 @@ functions:
       - stdc
     return_type: char *
     arguments:
-      - type: struct tm *
+      - type: const struct tm *
   - name: asctime_r
     standard: 
       - stdc
diff --git a/libc/newhdrgen/yaml_to_classes.py b/libc/newhdrgen/yaml_to_classes.py
index 6bccda8e03640..d88ba95ee3d6b 100644
--- a/libc/newhdrgen/yaml_to_classes.py
+++ b/libc/newhdrgen/yaml_to_classes.py
@@ -8,12 +8,11 @@
 #
 # ==-------------------------------------------------------------------------==#
 
-
 import yaml
 import argparse
-
 from pathlib import Path
 from header import HeaderFile
+from header2 import HeaderFile as HeaderFile2
 from class_implementation.classes.macro import Macro
 from class_implementation.classes.type import Type
 from class_implementation.classes.function import Function
@@ -22,18 +21,20 @@
 from class_implementation.classes.object import Object
 
 
-def yaml_to_classes(yaml_data):
+def yaml_to_classes(yaml_data, header_class, entry_points=None):
     """
     Convert YAML data to header classes.
 
     Args:
         yaml_data: The YAML data containing header specifications.
+        header_class: The class to use for creating the header (HeaderFile or HeaderFile2).
+        entry_points: A list of specific function names to include in the header.
 
     Returns:
         HeaderFile: An instance of HeaderFile populated with the data.
     """
     header_name = yaml_data.get("header")
-    header = HeaderFile(header_name)
+    header = header_class(header_name)
 
     for macro_data in yaml_data.get("macros", []):
         header.add_macro(Macro(macro_data["macro_name"], macro_data["macro_value"]))
@@ -49,12 +50,15 @@ def yaml_to_classes(yaml_data):
         )
 
     functions = yaml_data.get("functions", [])
+    if entry_points:
+        entry_points_set = set(entry_points)
+        functions = [f for f in functions if f["name"] in entry_points_set]
     sorted_functions = sorted(functions, key=lambda x: x["name"])
     guards = []
     guarded_function_dict = {}
     for function_data in sorted_functions:
         guard = function_data.get("guard", None)
-        if guard == None:
+        if guard is None:
             arguments = [arg["type"] for arg in function_data["arguments"]]
             attributes = function_data.get("attributes", None)
             standards = function_data.get("standards", None)
@@ -105,19 +109,21 @@ def yaml_to_classes(yaml_data):
     return header
 
 
-def load_yaml_file(yaml_file):
+def load_yaml_file(yaml_file, header_class, entry_points):
     """
     Load YAML file and convert it to header classes.
 
     Args:
-        yaml_file: The path to the YAML file.
+        yaml_file: Path to the YAML file.
+        header_class: The class to use for creating the header (HeaderFile or HeaderFile2).
+        entry_points: A list of specific function names to include in the header.
 
     Returns:
-        HeaderFile: An instance of HeaderFile populated with the data from the YAML file.
+        HeaderFile: An instance of HeaderFile populated with the data.
     """
     with open(yaml_file, "r") as f:
         yaml_data = yaml.safe_load(f)
-    return yaml_to_classes(yaml_data)
+    return yaml_to_classes(yaml_data, header_class, entry_points)
 
 
 def fill_public_api(header_str, h_def_content):
@@ -207,7 +213,14 @@ def increase_indent(self, flow=False, indentless=False):
     print(f"Added function {new_function.name} to {yaml_file}")
 
 
-def main(yaml_file, h_def_file, output_dir, add_function=None):
+def main(
+    yaml_file,
+    output_dir=None,
+    h_def_file=None,
+    add_function=None,
+    entry_points=None,
+    export_decls=False,
+):
     """
     Main function to generate header files from YAML and .h.def templates.
 
@@ -216,41 +229,50 @@ def main(yaml_file, h_def_file, output_dir, add_function=None):
         h_def_file: Path to the .h.def template file.
         output_dir: Directory to output the generated header file.
         add_function: Details of the function to be added to the YAML file (if any).
+        entry_points: A list of specific function names to include in the header.
+        export_decls: Flag to use HeaderFile2 for exporting declarations.
     """
-
     if add_function:
         add_function_to_yaml(yaml_file, add_function)
 
-    header = load_yaml_file(yaml_file)
-
-    with open(h_def_file, "r") as f:
-        h_def_content = f.read()
+    header_class = HeaderFile2 if export_decls else HeaderFile
+    header = load_yaml_file(yaml_file, header_class, entry_points)
 
     header_str = str(header)
-    final_header_content = fill_public_api(header_str, h_def_content)
 
-    output_file_name = Path(h_def_file).stem
-    output_file_path = Path(output_dir) / output_file_name
-
-    with open(output_file_path, "w") as f:
-        f.write(final_header_content)
+    if output_dir:
+        output_file_path = Path(output_dir)
+        if output_file_path.is_dir():
+            output_file_path /= f"{Path(yaml_file).stem}.h"
+    else:
+        output_file_path = Path(f"{Path(yaml_file).stem}.h")
+
+    if not export_decls and h_def_file:
+        with open(h_def_file, "r") as f:
+            h_def_content = f.read()
+        final_header_content = fill_public_api(header_str, h_def_content)
+        with open(output_file_path, "w") as f:
+            f.write(final_header_content)
+    else:
+        with open(output_file_path, "w") as f:
+            f.write(header_str)
 
     print(f"Generated header file: {output_file_path}")
 
 
 if __name__ == "__main__":
-    parser = argparse.ArgumentParser(
-        description="Generate header files from YAML and .h.def templates"
-    )
+    parser = argparse.ArgumentParser(description="Generate header files from YAML")
     parser.add_argument(
         "yaml_file", help="Path to the YAML file containing header specification"
     )
-    parser.add_argument("h_def_file", help="Path to the .h.def template file")
     parser.add_argument(
         "--output_dir",
-        default=".",
         help="Directory to output the generated header file",
     )
+    parser.add_argument(
+        "--h_def_file",
+        help="Path to the .h.def template file (required if not using --export_decls)",
+    )
     parser.add_argument(
         "--add_function",
         nargs=6,
@@ -264,6 +286,21 @@ def main(yaml_file, h_def_file, output_dir, add_function=None):
         ),
         help="Add a function to the YAML file",
     )
+    parser.add_argument(
+        "--e", action="append", help="Entry point to include", dest="entry_points"
+    )
+    parser.add_argument(
+        "--export-decls",
+        action="store_true",
+        help="Flag to use HeaderFile2 for exporting declarations",
+    )
     args = parser.parse_args()
 
-    main(args.yaml_file, args.h_def_file, args.output_dir, args.add_function)
+    main(
+        args.yaml_file,
+        args.output_dir,
+        args.h_def_file,
+        args.add_function,
+        args.entry_points,
+        args.export_decls,
+    )

@@ -0,0 +1,96 @@
#!/usr/bin/env python
#
# ===- HeaderFile Class for libc function headers -----------*- python -*--==#
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this file should have a name and license header that explain that it's for the GPU --export-decls version of headers.

@@ -13,33 +13,8 @@ types:
- type_name: __pthread_start_t
- type_name: __pthread_once_func_t
- type_name: __atfork_callback_t
enums:
- name: PTHREAD_CREATE_JOINABLE
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why did these get removed?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Was already in the h.def prior so we were duplicating enums.

content.append('extern "C" {')
content.append("#endif\n")

for include in self.includes:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

for the GPU version we don't need anything but the function and object declarations.

@@ -49,12 +50,15 @@ def yaml_to_classes(yaml_data):
)

functions = yaml_data.get("functions", [])
if entry_points:
entry_points_set = set(entry_points)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why is this set made?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The line after when checking for the names, sets have a faster operation time for checking because we are just seeing if the func name is in the set rather than scanning through all of entry_points everytime

Copy link
Contributor

@michaelrj-google michaelrj-google left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Overall looking good, just a few final changes

libc/newhdrgen/gpu_headers.py Outdated Show resolved Hide resolved
# ==-------------------------------------------------------------------------==#


class HeaderFile:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This class should have a different name from HeaderFile

from pathlib import Path
from header import HeaderFile
from gpu_headers import HeaderFile as HeaderFile2
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You should also use a more descriptive name than HeaderFile2 here.

Copy link
Contributor

@michaelrj-google michaelrj-google left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

- added entrypoints and headerfile parameters depending on target
- fixed nits in yaml files causing errors
- tested with new cmake config
- cmake patch will be seperate
- fixed NOEXCEPT to only be there for non-gpu
- added entrypoints and headerfile parameters depending on target
- fixed nits in yaml files causing errors
- tested with new cmake config
- cmake patch will be seperate
- fixed NOEXCEPT to only be there for non-gpu
@aaryanshukla aaryanshukla merged commit f1c957d into llvm:main Jul 16, 2024
3 of 5 checks passed
RoseZhang03 pushed a commit to aaryanshukla/llvm-project that referenced this pull request Jul 16, 2024
- added entrypoints and headerfile parameters depending on target
- fixed nits in yaml files causing errors
- tested with new cmake config
- cmake patch will be seperate
sgundapa pushed a commit to sgundapa/upstream_effort that referenced this pull request Jul 23, 2024
- added entrypoints and headerfile parameters depending on target
- fixed nits in yaml files causing errors
- tested with new cmake config
- cmake patch will be seperate
yuxuanchen1997 pushed a commit that referenced this pull request Jul 25, 2024
Summary:
- added entrypoints and headerfile parameters depending on target
- fixed nits in yaml files causing errors
- tested with new cmake config
- cmake patch will be seperate

Test Plan: 

Reviewers: 

Subscribers: 

Tasks: 

Tags: 


Differential Revision: https://phabricator.intern.facebook.com/D60251601
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants